The influence of data characteristics on interrater agreement among visual analysts.

Visual analysis is the primary method of analyzing single-case research data, yet relatively little is known about the variables that influence raters' decisions and rater agreement. Previous research has suggested that trend, variability, and autocorrelation may negatively affect interrater agreement, but studies have been limited by small numbers of graphs and participants whose knowledge of single-case research was not described. The purpose of this study was to examine the main and interaction effects of two values of each of six data characteristics (e.g., level, trend, and number of data points) on agreement among visual analysts. Using data from Lanovaz and Hranchuk (2021), we examined odds ratios to identify data characteristics that influence interrater agreement. Results suggest that trend and effect size, and to a lesser extent variability, have the largest effects on interrater agreement. We discuss the implications of our results for future research on improving interrater agreement among visual analysts.

A randomized controlled trial on the interconnected systems framework for school mental health and PBIS: Focus on proximal variables and school discipline.

This study reviews findings for the first randomized controlled trial (RCT) on the Interconnected Systems Framework (ISF) for school mental health (SMH) and Positive Behavioral Interventions and Supports (PBIS). Since its development in the late 2000s, the ISF has been supported by federally funded centers for SMH and PBIS, and, guided by a national workgroup, is being implemented in >50 communities in the United States. This experimental evaluation of the ISF involved an RCT implemented in 24 schools in two southeastern states, with the ISF implemented in eight schools, PBIS alone implemented in eight schools, and typically co-located PBIS+SMH implemented in eight schools. Related to very poor implementation, documented by two sources of fidelity data, two ISF schools were dropped from major analyses; hence, the study used a treatment on the treated (ToT; Rubin, 1974) as compared to a more traditional Intent-to-Treat approach (ITT; Lachin, 2000). This is the first paper from this large study, with emphasis here on proximal variables and school discipline. Within schools' multi-tiered systems of support (MTSS), ISF schools delivered more Tier 2 (early intervention) and Tier 3 (treatment) interventions to a greater proportion of students than the other two conditions by the second year of the intervention. There was also a dramatic difference in the provision of interventions by community mental health clinicians in ISF schools (almost half of interventions delivered) as compared to PBIS+SMH schools (around 3% of interventions delivered), underscoring the critical role of the ISF in integrating clinicians into MTSS teams and core school functions in SMH. As compared to the other two conditions, ISF schools also had reduced office discipline referrals (ODRs) and in-school suspensions, as well as reduced ODRs and out-of-school suspensions for African American students. Findings are discussed in relation to future directions of education-mental health system partnerships in improving the delivery and impact of SMH programs and services, demonstrated in the ISF.

An evaluation of the agreement between the conservative dual-criterion method and expert visual analysis.

The conservative dual-criterion (CDC) method was developed to standardize the analysis of single-subject experimental designs data, but to date its accuracy has been evaluated only by comparing results to the statistical parameters of graphs. Our study investigated agreement between expert visual analysts and the CDC method on 66 AB tiers from published multiple baseline graphs. We found strong agreement between the two methods for certain types of graphs and discuss implications of the findings and areas for future research.

Interrater Agreement on the Visual Analysis of Individual Tiers and Functional Relations in Multiple Baseline Designs.

Previous research on visual analysis has reported low levels of interrater agreement. However, many of these studies have methodological limitations (e.g., use of AB designs, undefined judgment task) that may have negatively influenced agreement. Our primary purpose was to evaluate whether agreement would be higher than previously reported if we addressed these weaknesses. Our secondary purposes were to investigate agreement at the tier level (i.e., the AB comparison) and at the functional relation level in multiple baseline designs and to examine the relationship between raters' decisions at each of these levels. We asked experts (N = 52) to make judgments about changes in the dependent variable in individual tiers and about the presence of an overall functional relation in 31 multiple baseline graphs. Our results indicate that interrater agreement was just at or just below minimally adequate levels for both types of decisions and that agreement at the individual tier level often resulted in agreement about the overall functional relation. We report additional findings and discuss implications for practice and future research.